Skip to content

feat(memory): episode lifecycle — dedup-on-write + eviction (cap/TTL)#17

Merged
jkyberneees merged 2 commits into
mainfrom
feat/memory-episode-lifecycle
Jun 6, 2026
Merged

feat(memory): episode lifecycle — dedup-on-write + eviction (cap/TTL)#17
jkyberneees merged 2 commits into
mainfrom
feat/memory-episode-lifecycle

Conversation

@jkyberneees
Copy link
Copy Markdown
Contributor

@jkyberneees jkyberneees commented Jun 6, 2026

Problem

EpisodeStore had no dedup and no eviction — episodes accumulated forever and near-duplicate sessions piled up unbounded (disk growth, noisier recall). This is the first slice of memory lifecycle (#6).

Change

Bounded, de-duplicated episode storage. Config-gated, safe defaults, no on-disk format change.

Dedup-on-write (newest-wins, provenance-gated)

A new episode whose cosine similarity to an existing one ≥ episode_dedup_threshold (default 0.92) replaces it. Uses an ephemeral RP embedder (the existing NewRPRanker primitive) over full on-disk summaries — deliberately not the shared dirty-rebuild vector index, avoiding a synchronous mid-write rebuild/re-entrancy. Provenance-safe: an untrusted near-dup can never evict a trusted/approved episode (trustRank mirrors the recall filter).

Eviction (prune-on-write)

Applies TTL (episode_ttl_days, default 0/off) then a count cap (max_episodes, default 500), deleting both the .md file and the index entry. Crash-safe order: files → writeIndex → markDirty (a crash leaves at most a dangling index entry, which rebuild/recall tolerate). Also exposes EpisodeStore.Prune() for session-end/CLI use.

Locking + a latent-bug fix

Dedup + .md write + index update + prune + markDirty now run under a single e.mu hold (writeLocked). This also fixes a latent bug where re-writing the same sessionID appended a duplicate index entry.

Config

EpisodeDedupThreshold / MaxEpisodes / EpisodeTTLDays wired through MemoryConfig, DefaultMemoryConfig, both overlay sites, and a new NewEpisodeStoreWithLifecycle. The bare NewEpisodeStore keeps lifecycle off, so existing callers/tests are unaffected. Documented in docs/CONFIG.md.

Deferred: fact supersession — facts have no per-entry metadata (true supersession needs a format change) and already get semantic dedup via merge-on-write + session-end LLM consolidation.

Tests

Dedup replace/threshold/disabled, provenance safety (both directions), eviction by count + TTL, TTL-disabled, self-overwrite regression, evicted-id absent from recall, -race concurrency (16 goroutines), config defaults + overlay-to-store wiring.

Verification

go build ./...                              # ok
go vet ./...                                # clean
gofmt -l internal/memory internal/config   # empty
go test ./internal/memory/... ./internal/config/... -race   # ok
go test ./... -short -race                  # PASS

🤖 Generated with Claude Code

jkyberneees and others added 2 commits June 6, 2026 14:12
@jkyberneees @claude
feat(memory): episode lifecycle — dedup-on-write + eviction (cap/TTL)
6b2baff 
EpisodeStore had no dedup and no eviction: episodes accumulated forever
and near-duplicate sessions piled up unbounded (disk growth, noisier
recall). This adds bounded, de-duplicated episode storage, config-gated
with safe defaults, no on-disk format change.

- Dedup-on-write: a new episode whose cosine similarity to an existing one
  is >= episode_dedup_threshold (default 0.92) REPLACES it (newest-wins).
  Uses an ephemeral RP embedder (the NewRPRanker primitive) over full
  on-disk summaries — never the shared dirty-rebuild vector index, avoiding
  mid-write re-entrancy. Provenance-gated: an untrusted near-dup can never
  evict a trusted/approved episode (trustRank mirrors the recall filter).
- Eviction: prune-on-write applies TTL (episode_ttl_days, default 0/off)
  then a count cap (max_episodes, default 500), deleting both the .md file
  and the index entry. Crash-safe order: files -> writeIndex -> markDirty.
  Also exposes EpisodeStore.Prune() for session-end/CLI use.
- Locking: dedup + .md write + index update + prune + markDirty now happen
  under a single e.mu hold (new writeLocked). Fixes a latent bug where
  re-writing the same sessionID appended a duplicate index entry.
- Config: EpisodeDedupThreshold / MaxEpisodes / EpisodeTTLDays wired through
  MemoryConfig, DefaultMemoryConfig, both overlay sites, and a new
  NewEpisodeStoreWithLifecycle (bare NewEpisodeStore keeps lifecycle off, so
  existing callers/tests are unaffected). Documented in docs/CONFIG.md.

Fact supersession is deferred (facts lack per-entry metadata; already
covered by merge-on-write + session-end LLM consolidation).

Tests: dedup replace/threshold/disabled, provenance safety (both
directions), eviction by count + TTL, TTL-disabled, self-overwrite
regression, evicted-id absent from recall, -race concurrency (16
goroutines), config defaults + overlay-to-store wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jkyberneees @claude
fix(memory): validate sessionID before deleting episode files
844da80 
Adversarial review (AI Verification Protocol) found a path-traversal
defense-in-depth gap: eviction/dedup called os.Remove on sessionIDs read
straight from index.json without validation, so a crafted/corrupted index
entry (e.g. "../victim") could delete a .md file OUTSIDE the episodes dir.
Every other file op in the package (Read/Write/Promote) already validates.

Add removeEpisodeFile(sessionID) which calls session.ValidateSessionID
before os.Remove, and route all three eviction/dedup deletions through it.
Adds a traversal-safe regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jkyberneees
Copy link
Copy Markdown
Contributor Author

jkyberneees commented Jun 6, 2026

🔍 AI Verification Protocol v5.2.7 — Certificate

Classification: NovelBehavior (new dedup/eviction logic) · LOC_filtered: ~587 — under the 1,500 standard-pipeline ceiling.

Adversarial pass (Agent D) — findings & fixes

ID Severity Finding Fix
D-01 🔴 Security (defense-in-depth) Eviction/dedup called os.Remove on sessionID read straight from index.json without validation. A crafted/corrupted index entry (e.g. "../victim") deleted a .md file outside the episodes dir — verified with a probe that deleted a file in the parent dir. Every other file op (Read/Write/Promote) already validates. New removeEpisodeFile(sessionID) calls session.ValidateSessionID before os.Remove; all three eviction/dedup deletions routed through it. Regression test TestEpisodeEviction_TraversalSafe. Fixed in 844da80.

Probes that passed (no fix needed):

  • Provenance-skip leaves the trusted .md intact.
  • No index/file orphans after dedup chains, dedup+cap, or 6 sequential near-dup replacements.
  • Empty/whitespace summary doesn't crash (added as distinct; excluded from the vector index by the existing empty-text skip).

Axes

Axis Result Evidence
2.1 Semantic correctness ✅ Dedup replace/threshold/disabled, eviction count+TTL, self-overwrite all tested; behaviors match contract.
2.3 Security surface ✅ (after D-01) Path-traversal in file deletion closed; os.Remove now validated like the rest of the package.
2.4 Structural integrity ✅ Single e.mu hold over dedup→write→index→prune→markDirty; crash-safe file→index→markDirty ordering; ephemeral RP avoids shared dirty-rebuild re-entrancy.
2.5 Behavioral exploration ✅ Adversarial probes (traversal, provenance-skip, dedup+cap, orphans, empty summary) + 16-goroutine -race.
2.9 Documentation ✅ New exported NewEpisodeStoreWithLifecycle/Prune carry godoc; 3 config keys documented in docs/CONFIG.md. No removed/changed public signatures (NewEpisodeStore unchanged).

Signals

go build ✅ · go vet ✅ · gofmt -l empty ✅ · go test ./internal/memory/... ./internal/config/... -race ✅ · go test ./... -short -race ✅.

Verdict: AutoApprove after remediation. The one finding (D-01) is fixed and regression-tested before merge.

@jkyberneees jkyberneees merged commit 6d2cecc into main Jun 6, 2026
6 checks passed
@jkyberneees jkyberneees deleted the feat/memory-episode-lifecycle branch June 6, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jkyberneees